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ABSTRACT 

There is a significant gap between theories of general 
psychological functions on one hand (e.g., memory) and theories of 
mathematical content knowledge on the other (e.g., content of algebra). To 
better guide the design of ground breaking and demonstrably better 
mathematics instruction, we need instructional principles and associated 
design methods to fill this gap in a way that is not only consistent with 
psychological and content theories but prompts and guides us beyond what 
those theories can do. Toward this goal, I reflect on lessons from past and 
current Cognitive Tutor mathematics projects. From this experience, I have 
abstracted four instructional bridging principles, Situation-Abstraction, 
Action-Generalization, Visual-Verbal, and Conceptual-Procedural , and 
associated methods for applying them. I illustrate these in the context of 
the design of the successful Cognitive Tutor Algebra course (now in more than 
800 schools) and the on-going research and development of a Cognitive Tutor 
course for 6th grade mathematics. (Author) 
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TOWARD EVIDENCE FOR INSTRUCTIONAL DESIGN 
PRINCIPLES: EXAMPLES FROM 
COGNITIVE TUTOR MATH 6 

Kenneth R. Koedinger 
Carnegie Mellon University 
Koedinger@cmu.edu 

There is a significant gap between theories of general psychological functions on 
one hand (e.g., memory) and theories of mathematical content knowledge on the 
other (e.g., content of algebra). To better guide the design of ground breaking and 
demonstrably better mathematics instruction, we need instructional principles and 
associated design methods to fill this gap in a way that is not only consistent with 
psychological and content theories but prompts and guides us beyond what those 
theories can do. Toward this goal, I reflect on lessons from past and current Cognitive 
Thtor mathematics projects. From this experience, I have abstracted four instructional 
bridging principles, Situation- Abstraction, Action-Generalization, Visual -Verbal, and 
Conceptual-Procedural, and associated methods for applying them. I illustrate these 
in the context of the design of the successful Cognitive Tutor Algebra course (now in 
more than 800 schools) and the on-going research and development of a Cognitive 
Tutor course for b** grade mathematics. 

Introduction 



My first goal in this paper is to summarize a successful educational innovation 
grounded in psychological research on mathematical learners and learning. Cogni- 
tive Tutor Algebra is a full year course that combines a problem-based consumable 
textbook, used 3 days a week, and intelligent tutoring software, used 2 days a week. 
Cognitive Tutor Algebra has been successful commercially, sold to over 800 schools, 
and pedagogically, shown to yield higher student achievement than alternatives. My 
second goal is to summarize aspects of the psychological theory, ACT-R, that sup- 
ported the design of this course and helps explain its apparent success. My third goal 
is to motivate the need for theory at an intermediate level between general cognitive 
theories, like ACT-R, and models specific to narrow content areas, like proportional 
reasoning. Such theory is needed to fill the gap that exists between general theory and 
instructional innovations, so that fewer unguided, purely intuitive decisions need to 
be made in instructional design. I will discuss four principles to guide such decisions 
and describe domain-specific methods to implement these principles. My fourth and 
final goal is to introduce a new Cognitive Tutor mathematics course for grade and 
to illustrate the four principles using text and tutor lessons from the on-going course 
development. 



Success of Cognitive I\itors and Humility Message 



Before discussing the Cognitive Tutor Algebra project, let me start with some 
background. Following an experience creating and field testing an intelligent tutoring 
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system for geometry proof (Koedinger & Anderson, 1993), in which I was one of four 
teachers involved, it became particularly clear to me how difficult and important it is 
to attend to the social context of an educational innovation (cf., Lehrer, Randle, & San- 
cillo, 1989; Schofield, Evans-Rhodes, & Huber, 1990). Two key features of the social 
context are integrating the innovation with other aspects of the curriculum and innova- 
tion-specific teacher professional development. When we started the algebra project 
in 1992, we took what we called a “client-centered approach” and sought guidance 
from mathematics educators toward meeting national standards (Koedinger, Ander- 
son, Hadley, & Mark, 1997). Perhaps most importantly, we decided that to address the 
curriculum integration challenge, we would create a full course including both the text 
materials (used 3 days a week) and the technology (used 2 days a week). The math- 
ematics curriculum supervisor for the Pittsburgh Public Schools, Diane Briars, pointed 
us to teacher Bill Hadley who had already been writing new algebra text particularly 
focused on helping students make sense of algebra. We worked together, combining 
classroom intuition and cognitive science, to produce a problem-based course that 
connects multiple representations of functions and employs advanced computational 
tools. Problems and projects in both the text and software connect to real-world uses of 
algebraic reasoning like estimating the cost of a rental car, choosing between long-dis- 
tance phone services, predicting the decline of the condor population, planning profits 
for shoveling snow, comparing the current quantity and growth rate of old growth 
forest in the US to the harvest rate. 

In three full-year multi-school field studies, we demonstrated that students in 
a Cognitive Tutor Algebra course learn more than students in comparison classes 
both on assessments of problem solving and representation use and on standardized 
assessments of basic skills (Koedinger et al., 1997; Koedinger, Corbett, Ritter, & Sha- 
piro, 2000). As the dissemination of the algebra course was ramping up, in 1995 we 
obtained funding from local foundations (Heinz, Buhl, Grable, Mellon, and Pittsburgh 
Foundations) to create full courses for geometry (led by myself) and algebra 2 (led 
by Albert Corbett). With the help of Carnegie Mellon’s technology transfer office, 
in 1998 we created Carnegie Learning, Inc to further develop and market all three 
courses. The number of schools using Cognitive Tutor Algebra has roughly doubled 
each year and is now over 800. 

Such success is encouraging, but the job is not done. Education is a very hard 
problem. Besides tough issues of values, culture, race, politics, respect for teachers, 
testing, etc., we certainly have not cracked how the mind works or how learning is 
best achieved. We need to remain humble. We need to recognize that our instructional 
beliefs, practices, theories may be wrong. We need to put them to the test and be proud 
of failure. In the context of hypotheses and principles, failure advances theory. Given 
the difficulty of the education problem, we also need to collaborate. No one can suc- 
cessfully tackle a significant educational problem alone. We need to combine expertise 
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from multiple domains, particularly education researchers, cognitive psychologists 
and learning scientists, but also anthropologists, computer scientists, measurement, 
policy experts, etc. 

In the remainder of this paper, I reflect upon the lessons learned in our Cognitive 
Tutor projects particularly at the level of principles and design methods. I illustrate 
these primarily with examples from the on-going research and development of the 
Cognitive Tutor course for 6“’ grade mathematics. In 1999, with support from Carnegie 
Learning, we began a project to create full Cognitive Tutor courses for middle school 
mathematics with Albert Corbett leading the 7^*" and 8^ grade course development and 
myself leading the 6**" grade course development. 

Theory, Principles, and Methods behind the Design of Cognitive Ibtors 
ACT-R and General Principles for Instructional Design 

Cognitive Tutors are based on the ACT-R theory of learning and performance 
(Anderson & Lebiere, 1998). The theory distinguishes between tacit performance 
knowledge, so-called “procedural knowledge” and static verbalizable knowledge, 
so-called “declarative knowledge”. According to ACT-R, performance knowledge can 
only be learned by doing, not by listening or watching. In other words, it is induced 
from constructive experiences — it cannot be directly placed in our heads. Such per- 
formance knowledge is represented in the notation of if-then production rules that 
associate internal goals and/or external perceptual cues with new internal goals and/or 
external actions. Three examples of English versions of production rules are shown in 
Table 1, which is discussed below. 

Developing Cognitive Tutor software involves the use of the ACT-R theory and 
empirical studies of learners to create a “cognitive model”. A cognitive model uses a 
production system to represent the multiple strategies students might employ as well 
as their typical student misconceptions. The following provides a simplified example 
from algebra equation where these three production rules are alternative ways to 
respond to the same goal (a, b, c, and d are any numbers and x is any variable to be 
solved for): 

Strategy 1: IF the goal is to solve a(bjc+c) = d 

THEN rewrite this as bjc + c = d/a 

Strategy 2: IF the goal is to solve a(bjc+c) = d 

THEN rewrite this as abjc + ac = d 

Misconception: IF the goal is to solve a(bjc+c) = d 

THEN rewrite this as abjc + c = d 

The cognitive model is used with an algorithm called model tracing to follow stu- 
dents through their individual approach a problem. In so doing, the tutor is able to pro- 
vide context-sensitive instruction. The cognitive model is also used by an algorithm 
called knowledge tracing that assesses students’ knowledge growth as they succeed or 
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fail on actions associated with the production rules in the cognitive model. The results 
of knowledge tracing are displayed with “skill bars” in the computer tutor interface 
(see the top right in Figure 5) and are used to select activities and adapt pacing to indi- 
vidual student needs. The cognitive model is not only the key workhouse within the 
automated Cognitive Tutors, but it is also a theoretical tool used to guide the design 
of other aspects of instruction including problem and interface design, text materials, 
and classroom activities. 

In 1995, we published a report on the status of the lessons learned to date from 
Cognitive Tutor development (Anderson, Corbett, Koedinger, & Pelletier, 1995). We 
described some general Cognitive Tutor design principles consistent with ACT-R and 
our research and development experience to that date. A key principle recommended 
to “represent student competence as a production set”. In other words, to base instruc- 
tion on an analysis not of mathematical content per se, but of the way in which stu- 
dents think about the content. The notion of “theorems in action” (Nunes, Schliemann, 
& Carraher, 1993; Vergnaud, 1982) is very similar to production rules. 

The idea here is to characterize how students may think about mathematics dif- 
ferently (“informally” or “intuitively”. Resnick, 1987) than is normatively taught or 
present in textbooks (e.g., see production #1 in Table 1). In addition, production rules 
represent not just how students think about mathematics operations, but when they 
retrieve relevant knowledge. This feature of production rules is critical to representing 
the fact that the knowledge students acquire is sometimes overly specific, so it does 
not transfer well, and is sometimes overly general, so it leads to errors. In other words, 
students’ “theorems in action” are often more specific than actual theorems or rules. 
For instance, students can combine like terms in an equation when coefficients are 
present (e.g., 2x-\-3x => 5a:) but not when a coefficient is missing (e.g., x - 0.2jc). See 
production #2 in Table 1. Alternatively, students’ theorems in action are often more 
general than actual theorems or rules. For instance, they may learn to combine num- 
bers by the operator between them (e.g., 2*3 + 4= j:=> 6-1-4 = jc) without acquiring 
knowledge that prevents order of operations errors (e.g.,jc*3-i-4 = 10=>jc*7 = 10). 
See production #3 in Table 1 . It should be clear that the “rules of mathematics” and the 
“rules of mathematical thinking” are not the same. 

A fundamental assumption of ACT-R is that people learn by doing as the brain 
generalizes from one’s explicit and implicit interpretations or “encodings” of one’s 
experiences. This assumption is, in my opinion, quite consistent with constructivism. 
It is not the information or even the instructional activities students are given per se 
that matters, but how students experience and engage in such information and activi- 
ties that determines what knowledge they construct from them. Thus, another principle 
in Anderson et al. (1995) was “provide instruction in the problem-solving context”. 

The strength of ACT-R and the principles summarized in Anderson et al. (1995) 
are that they are general and can apply in multiple domains. However, this strength 
of generality comes with a limitation. An instructional designer can easily apply the 
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Table 1. Example Production Rules 




Production Rules in English 


Example of its application 



1. Production acquired implicitly 
(not explicitly taught) 

IF you want to find Unknown and the 
inal result is Known-Result and the last 
step was to apply Last-Operator to Last-Num, 
THEN work backwards by inverting 
Last-Operator and applying it to Known-Result 
and Last-Num 


Part of informal “unwind” 
strategy that solves problems of 
the form “3x + 48 = 63” 
without by working backwards: 
63-48-> 15/3 = 5 


2. Overly specific production 

IF “ax + bx” appears in an expression 

and c = a + b 

THEN replace it with “cx” 


Works for “2x + 3x” but not for 
“X + 3x” 


3. Overly general production 

IF “Numl + Num2” appears in an expression 

THEN replace it with the sum 


Leads to order of operations 
error: “x * 3 + 4” is rewritten 
as “x * 7” 
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theory and principles in multiple ways, some good, some OK, and some bad. For 
instance, the principle “minimize working memory load” can be implemented in many 
ways including having the technology perform trivial operations for students. How- 
ever, this approach will only be effective if the designer is right about what is trivial. 
Getting the application of the theory and principles right depends on getting domain- 
specific details right. Thus, a general theme of this paper is that we not only need to 
make progress in better articulating theory and principles, but also in specifying asso^ 
dated methods that better ensure these principles will be appropriately applied. This 
point applies as well to other proposals for instructional principles. 

Instructional Design Principles require Empirical Methods to Successfully 
Implement 

The National Research Council’s volume How People Learn (Bransford, Brown, 
& Cocking, 1999) proposes three general instructional principles: 1) build on prior 
knowledge, 2) connect learning of facts and procedures with conceptual learning, and 
3) support meta-cognition. These are common to many other approaches. To take the 
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first, for instance, the NCTM states: “Students must learn mathematics with under- 
standing, actively building new knowledge from experience and prior knowledge,” 
(NCTM, 2000, p. 16). Romberg & de Lange (2002) describe the Realistic Mathemat- 
ics Education approach as recommending “making use of students informal math- 
ematical activity to support their development of more formal strategies”. 

But, what prior, informal knowledge do students have? Sometimes theoretical 
analysis is used to predict prior knowledge under the assumption that smaller com- 
ponent tasks are more likely to tap prior knowledge than larger whole tasks (cf.. Van 
Merrienboer, 1997). Such analysis focuses on mathematical content and its external 
forms. However, smaller tasks are not always simpler. It is not the surface form of 
tasks or external representations that determine how accessible they are to students. 
Instead, it is the internal mental representations that students acquire and use in task 
performance that determines what will be simple or not. Thus, to identify what prior 
knowledge students have and whether task X is going to be more likely than task Y 
to tap prior knowledge, it is not sufficient to analyze the content domain. Instead it is 
critical to study how students actually perform on tasks - to see student thinking as it 
really is, not as a content analysis might assume it to be. 

Results of a Difficulty Factors Assessment indicating that ^ ^smaller is not 

always simpler’’ 

Consider the three problems shown below, a story problem, word problem, and 
equation, all with the same underlying quantitative structure and the same solution. 

Story Problem: As a waiter, Ted gets $6 per hour. One night he made $66 in 

tips and earned a total of $81.90. How many hours did Ted work? 

Word Problem: Starting with some number, if I multiply it by 6 and then 

add 66, 1 get 81 .90. What number did I start with? 

Equation: x*6 + 66 = 81 .90 

Which would be most difficult for high school students in a first year algebra 
course? Nathan & Koedinger (2000) discussed results of surveys of teachers and 
mathematics education researchers on a variation of this question. The survey respon- 
dents tended to predict that such story problems would be most difficult and such 
equations would be easiest. Typical justifications for this prediction include that the 
story problem requires more reading or that the way the story problem is solved is by 
translating to the equation. 

In contrast, what we found in the two original studies (Koedinger & Nathan, in 
press) and in subsequent replications is that students perform best at the story and 
word problems (70% and 61% respectively) like these and worst at the analogous 
equations (42%). Clearly many students were not solving the story and word problems 
using equation solving. Instead they used alternative informal strategies like guess- 
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and-test and “unwinding”, working backwards from the result, inverting operations to 
find the unknown starting quantity (cf., Bednarz, Kieran, & Lee, 1996). Students had 
difficulty in comprehending equations and, when they did succeed in comprehending, 
they often had difficulty in reliably executing the equation solving strategy. 

This result is important within the algebra domain. It indicates that if we want 
to create instruction that builds on prior knowledge, we should make use of the fact 
that beginning algebra students have quantitative reasoning skills that can be tapped 
through verbal or situational contexts. Unlike many textbooks that teach equation 
solving prior to story problem solving, it may be better to use story problem situations 
and verbal descriptions first to help students understand quantitative relationships with 
them before moving to more abstract processing. 

However, there is also a more general message. In order to apply principles like 
“build on prior knowledge” we cannot assume that it is obvious what prior knowl- 
edge students’ posses. Instead, we need to do empirical studies to assess what prior 
knowledge students have and what forms of presentation best elicit this knowledge. 
To remind us of this message, we need to repeat the mantra: “The student is not like 
me”. 

Cognitive T\itors are Developed Using the Design Experiment Methodology 

Assessment experiments, like the one above, that compare the effect of various 
difficulty factors on student performance are a common component in my instructional 
design approach. Such “Difficulty Factors Assessments” are one part of my version of 
the “design experiment” (Brown, 1992) approach. Design experiments involve theo- 
retically motivated cycles of design, feedback, and redesign. While use of theory is 
critical to the approach, design experiments do not involve a linear process of applying 
theory to create innovations that are then tested. The approach is neither purely basic 
nor purely applied, but combines the goals of both. It is between “the hare of intui- 
tive design and the tortoise of cumulative science”. In addition to theory, we rely on 
intuitions of multiple team members, particularly practicing teachers to help generate 
design ideas. We test and prune ideas in embedded cycles at different time scales (see 
Table 2). Some cycles are short, like a 5-minute assessment to test a prior knowledge 
hypothesis or feedback from a project teacher on how a lesson went today and how 
might it be improved for tomorrow. Some cycles are at the curriculum unit (or chap- 
ter) level, like investigating how much a unit improved learning from pre- to post-test 
relative to an alternative approach. A typical “parametric” study compares different 
versions of a Cognitive Tutor that vary on a key dimension of interest. The longest 
cycles are at the full course level, comparing student end-of-course performance with 
that resulting from alternative courses. 

Instructional Bridging Principles 

I now describe four “instructional bridging” principles that are more specific, 
particularly to mathematics, than the more general principles from ACT-R and others 
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Table 2. Empirical Methods at Different Temporal Grain Sizes 



1. Single point assessment of performance. Paper quiz or informal observation 
(minutes). 

2. Lab study or class pull-out. Pre-Post lesson length comparison (day) 

3. Replacement unit “parametric” study. Pre-Post unit length comparison (week/ 
month) 

4. Full course field study. Pre-Post course length comparison (semester/year) 

3. Instructional Bridging Principles 



described above. I have abstracted these principles from our experience developing 
Cognitive Tutor courses, from associated observations and experiments, and from the 
mathematics education and cognitive and developmental psychology literatures. In 
this section, I introduce the four principles and the underlying motivation for them. In 
the next section, I use examples from the 6^^" grade Cognitive Tutor course my team is 
developing to illustrate an on-going application of these principles. 

The four “instructional bridging” principles are: 

1. Situation-Abstraction: Bridge from concrete situational to abstract symbolic 
representations. 

2. Action-Generalization: Bridge from doing with instances to explaining with 
generalizations. 

3. Visual-Verbal: Integrate pictorial and verbal/symbolic representations. 

4. Conceptual-Procedural: Integrate conceptual and procedural instruction. 

These are specific variations on more general principles like “build on prior knowl- 
edge” and “connect and integrate multiple representations of knowledge”. 

Different kinds of evidence for these principles can be associated with the differ- 
ent kinds of studies in Table 2. One heuristic form of evidence (#1 in Table 2) assesses 
students’ performance to see whether one form of activity/representation is closer 
to students’ prior knowledge and thus easier. Another stronger form of evidence is 
instructional studies (#2 or #3) where an experimental comparison is made between 
instruction based on the principle and some other plausible form of instruction. A third 
less strong form of evidence is full course field studies (#4) where the experimental 
curriculum is consistent with the principle in question, but the comparison curriculum 
is not. The limitation of full course studies is that experimental and comparison cur- 
ricula inevitably differ in many more ways than whether they are consistent or not with 
the principle in question. 
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Situation- Abstraction Principle: Bridge from Concrete Situational to Abstract 
Symbolic 

The “Situation- Abstraction” principle is a consistent with one of the key features 
of the Cognitive Tutor Algebra course. It recommends instruction that uses more 
familiar concrete problem situations and descriptions as a bridge to powerful abstract 
symbolic forms. A key premise is that for many ideas and concepts, we can find con- 
crete situations or verbal descriptions that use or communicate these ideas in ways that 
are more comprehensible and accessible to students than more abstract symbolic and 
conventionalized forms of mathematical expression. Of course, these conventional 
symbolic forms are crucial to powerful thinking and communication, particularly in 
math and science and so are critical goals of instruction. The key hypothesis is that 
robust competence with abstract symbols can be reached more effectively and effi- 
ciently by employing this Situation- Abstraction principle. 

This principle is similar or even identical to recommendations of other approaches 
(e.g., Nunes, Schliemann, & Carraher, 1993; Collins, Brown, & Newman, 1989; Cog- 
nition and Technology Group (CTG), 1997). For instance, the Realistic Mathematics 
Education approach recommends “developing instruction based in experientially real 
contexts” (Romberg & de Lange, 2002, p. 9). This approach has been nicely imple- 
mented in the Math in Context curriculum (National Center for Research in Math- 
^ ematical Sciences Education & Freudenthal Institute, 1 997“ 1998). ^ 

Evidence for this principle includes of a number of full-year field studies (CTG, 

1997; Koedinger et al, 1997) and some unit-level studies (Brenner, Mayer, Moseley, 

Brar, Duran, Reed, & Webb, 1997; Nathan, Stephens, Masarik, Alibali, & Koedinger 
2002). However, all these studies involve substantial curricula that vary in other ways 
from the comparison curricula (e.g., use of multiple representations). The theoretical 
rationales for the benefits of early use of concrete or authentic situations include better 
connection with student prior knowledge, facilitating student motivation, facilitating 
memory, and facilitating transfer to authentic real world goals. We need more studies 
to test these different underlying rationales. My colleagues and I have emphasized 
using situations to connect with prior knowledge and demonstrated, as described 
above, that certain problem situations are easier for students to understand than cor- 
responding symbolic problems. However, we have not “closed the loop” in a scientifi- 
cally rigorous way, for instance, by showing that instruction that introduces situations 
before abstractions leads to better learning than instruction that introduces abstractions 
before situations but is otherwise the same. 

While this principle is well known, it is not necessarily easy to implement effec- 
tively in the classroom as Deborah Ball (1993) noted: 

How do I (as a mathematics teacher) create experiences for my students that 
connect with what they now know and care about but that also transcend 
the present? How do I value their interests and also connect them to ideas 
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and traditions growing out of centuries of mathematical exploration and 

invention? (p. 375) 

We should not expect that all kinds of situational support are valuable in all 
domains. Whether and what kinds of situational supports make sense or are interesting 
to students depends on the domain - a point I will illustrate below with data from the 
b'*' grade Cognitive Tutor project. 

Acdon-Generalizadon Prindple: Bridge from Doing with Instances to Explain- 
ing with Generalizations 

The “Action-Generalization” principle is also consistent with the Cognitive Tutor 
Algebra course. It recommends instruction that has students engaging in problem 
solving (doing) with specific instances of a mathematical relationship as a bridge to 
explaining that relationship in general terms. For instance, we might have students 
solve a problem for specific numerical instances (e.g., 2*4 = 8 and 3*4=12) before 
having them explain what they are doing in a general form (e.g., jc * 4; cf., Bednarz, 
Kieran, & Lee, 1996). Or, we might have students measure and add the angles in 
specific triangles before discovering the generalization and explaining it in their own 
words. 

In the process of designing the Algebra Cognitive Tutor, we performed a para- 
metric study in which we compared tutor versions with different orderings of ques- 
tions about the problem situation (Koedinger & Anderson, 1998). One tutor version 
was modeled after a popular algebra textbook (Forester, 1984). Here students were 
asked first to define an independent variable and write an expression for the dependent 
quantity then asked to solve for multiple instances of the variable. Another tutor ver- 
sion was consistent with the Action-Generalization principle. In this version, students 
were first asked to solve the problem situation for a couple specific instances and then 
asked to write an expression for the dependent quantity. Students showed significantly 
greater pre-to-post learning gain in the version in which instances preceded the general 
expression, consistent with the Action-Generalization principle. 

The Action-Generalization principle is a variation on the Realistic Mathematics 
Education notion of progressive formalization. Hans Freudenthal (1983) believed that 
“students are entitled to recapitulate in a fashion the learning process of mankind” 
(p. ix). This principle is also related to other recommendations for promoting student 
meta-cognition (e.g., Bransford et al., 1999), self-reflection, and particularly self- 
explanation (e.g., Chi, de Leeuw, Chiu, & Lavancher, 1994). We performed a series of 
parametric studies with the Geometry Cognitive Tutor (Aleven & Koedinger, 2002) 
comparing a textbook-like version in which students solved for measures in given 
figures with a self-explanation version in which students explained their problem- 
solving steps by indicating the relevant geometry rule. These experiments demon- 
strated a “less-is-more” benefit for self-explanation whereby students learn deeper. 
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more transferable knowledge from self-explaining even though the extra time needed 
for explanation means they solve fewer problems in the same instructional period. 
These experiments also demonstrated that such benefits can be achieved with a simple 
approach to explanation (referencing rules in a glossary) that is easily implemented 
in computer software. We are currently making progress on the more difficult task of 
having the software give feedback on student explanation written in their own words 
(Aleven, Popescu, Koedinger, 2002). 

Both the algebra and geometry studies were motivated, at least in part, by results 
of Difficulty Factors Assessments. We had found that students were more successful 
at problem solving with specific instances than at explaining in general terms both in 
algebra (Heffernan & Koedinger, 1998) and geometry (Aleven & Koedinger, 2002). 
Unlike the evidence for the Situation-Abstraction principle, the evidence for the 
Action-Generalization principle fairly nicely “closes the loop” in the following sense. 
First, the two parametric studies show how a result from a single point assessment 
can be used to guide the application of an instructional principle to create a novel 
instructional design. Second, they show that the students learn more from an instruc- 
tional design consistent with the Action-Generalization principle than from one that is 
otherwise the same, but is not consistent with the principle. 

Visual- Verbal Principle: Integrate Pictorial and Verbal Representations 

The Visual-Verbal principle recommends instruction that helps students integrate 
visual, spatial, or analog representations of an idea with verbal, sequential, or digital 
representations of that idea. In Cognitive Tlitor Algebra, the use of multiple represen- 
tations of functions, the more visual graph and table representations and the verbal- 
symbolic situation and equation representations, is consistent with this principle. 
However, it was not generally or explicitly applied. This principle is an expression of 
a major theme of the work of Robbie Case and colleagues (Griffin, Case, & Siegler, 
1994; Kalchman, Moss, & Case, 2001). Case observed that students typically come 
to a new idea both with relevant visual intuitions (e.g., sense of size of quantities) and 
relevant verbal knowledge (e.g., knowing the counting sequence: one, two, three...). 
He suggested that deeper understanding and competence comes from a careful inte- 
gration of this visual and verbal knowledge into a “central conceptual structure”. 
Case and colleagues have successfully employed this principle in at least three areas, 
early number, rational number, and functions (Kalchman, Moss, & Case, 2001). Typi- 
cal results show students in experimental conditions with poorer backgrounds (e.g., 
low SES 4 year olds) learning more than students in control conditions with similar 
backgrounds and catching up or exceeding students in control conditions with better 
backgrounds (e.g., high SES 4 year olds). 

Another influence on this principle is the Singapore Primary Mathematics text- 
book series (Singapore Ministry of Education, 1999), the preface of which states: 
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The main feature of the package is the use Concrete => Pictorial => Abstract 
approach. The pupils are provided with the necessary learning experiences 
beginning with the concrete and pictorial stages, followed by the abstract 
stage to enable them to learn mathematics meaningfully, (p. 3) 

This statement and the associated materials combine (and confound) aspects of the 
Situation-Abstraction, Action-Explanation, and Visual-Verbal principles. However, 
visual “pictorial” representations are clearly emphasized. 

Conceptual-Procedural Principle: Integrate Conceptual and Procedural 
Instruction 

The Conceptual-Procedural principle recommends instruction that supports 
learning of both conceptual and procedural knowledge and facilitates students in 
integrating the two. Rittle-Johnson & Siegler (1998) define conceptual knowledge as 
the understanding of principles and of relations between pieces of knowledge that are 
needed to solve novel tasks. They define procedural knowledge as the step-by-step 
actions (algorithms) efficient solving of routine problem-solving tasks. Many other 
researchers have addressed this distinction (e.g., Hiebert, 1986; Lesh & Landau, 1983; 
Ma, 1999; Starr, 2000) and there has been considerable debate about which kind of 
knowledge comes first developmentally and which kind should be instructed first. 
Some advocate that students learn concepts first and should use that knowledge to 
generate and select procedures (e.g., Gelman & Williams, 1998; Hiebert & Wearne, 
1996). Others advocate that students learn procedures first and then should extract 
domain concepts from that experience (e.g., Hiebert & Wearne, 1996; Siegler, 1991). 
It may well be that different orderings are appropriate for different mathematical ideas. 
More importantly, it may be better to focus on how instruction can integrate the two 
in an iterative process. In such an approach, increases in one type of knowledge may 
lead to increases in the other type, which leads to further increases in the first (Rittle- 
Johnson & Alibali, 1999). 

While many will acknowledge the effectiveness of Cognitive Tutors for proce- 
dural knowledge acquisition, the question arises about whether they are effective in 
supporting student acquisition of conceptual knowledge. Our work on Cognitive Tutor 
Math 6 has attempted to address this question (Rittle-Johnson & Koedinger, 2001; 
2002 ). 

Using the Bridging Principles in the Design of Cognitive l\itor Math 6 
Course Overview 

We started work on the middle school text and software in the fall of 1999. In 
2001-2002 school year the 6^*^ grade course included 9 units of text and 36 units of 
software, summarized in Table 3. To illustrate the use of the principles, I will focus on 
the algebra and rational number strands running through the curriculum. 
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Table 3. Cognitive Tutor Math 6 Course Content — 2001-02 School Year 



Textbook Units 


Tutor Units 


Strands 


1 . Finding Patterns & 
Writing Rules 


Algebraic Expression Generation, 
Expression Evaluation, Picture Algebra 


Algebra 


2. Grounding 
Place Value in 
Measurement 


Decimal Number Line, Decimal Place 
Value, Decimal Addition & Subtraction 


Number 


3. Grounding 

Decimals in Area & 
Perimeter 


Area & Perimeter with Decimals, 
Picture Algebra with Multiplication 


Number 

Geometry 


4. Understanding 
Division 


Expr Gen with Div, Compatible 
Numbers, Simplifying Div, Decimal Div 
with Area 


Number 

Algebra 


5. Understanding 
Fractions, 
Equivalence, 
Addition & 
Subtraction 


Fraction Concept, Proportional 
Reasoning, Probability, Fraction 
Arithmetic, 

Picture Algebra with Fractions 


Number 
Data & 
Probability 


6. Factors & Multiples 


Common Factors, Common Multiples, 
Scaling, Proportionality with Factors & 
Multiples 


Number 


7. Fraction 

Multiplication & 
Division 


Fraction Mult with Area, Picture 
Division, Fraction Mult & Div with Area 


Number 

Geometry 


8. Data Analysis 


Simple Histogram & Central Tendency 


Data 


9. Properties of Angles 
& Triangles 


Sorting & Defining, Drawing, 
Discovering Properties, Angle Solving, 
Similar Triangles 


Geometry 



Emphasixing the need for content-relevant data. 

Instructional principles are true or useful only to the extent that the application of 
them is guided by content-relevant data. Our research related to the Situation- Abstrac- 
tion principle provides an example. Table 4 illustrates different content areas in middle 
school math where we compared concrete story problem situations with abstract con- 
text-free problems. The table shows 6^ graders’ average percent correct on multiple 
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pre-test items in each category. In three of the areas, the problem situation consistently 
facilitates performance significantly above the abstract problem. These are decimal 
place value and decimal arithmetic (Rittle-Johnson & Koedinger, 2002) and fraction 
addition (Rittle-Johnson & Koedinger, 2001a). In data analysis (cf., Baker, Corbett, & 
Koedinger, 2001), the situation facilitated performance on a global interpretation task, 
but not on a local interpretation task. In the area of factors and multiples, the situation 
reduced performance. 

Application of the Situation- Abstraction principle may not be effective for con- 
cepts and procedures related to factors and multiples unless situations can be found 
that are easier to understand than abstract problems. While one might still want to 
use such a problem situation as motivation for learning, given this data, it does not 
appear that such a situation will provide a student with easier or more direct access to 
understanding. 

Examples from the Algebra Strand of Cognitive Hi tor Math 6 

Unit 1 of the grade curriculum is “Finding Patterns and Writing Rules”. The 
curriculum starts with a pre-algebra unit because it provides a good introduction to 
pattern finding, induction, and explaining patterns in general terms. This unit thus sets 
a tone for the rest of the school year where students will engage in pattern finding and 
rule discovery to gain deeper access into new concepts. 

Pattern induction and rule writing lessons illustrating Situation-Abstrac- 
tion and Action-Generalization. 

Figure 1 shows a text sample from the first lesson in Unit 1, which is intended 
as a fun opening activity. Students engage in both a simple pattern discovery, find the 
minimum number of sections (pieces of cake) for N cuts (N+1), and a difficult pattern 
discovery, find the maximum number of sections for N cuts (left to reader!). Students 
achieve various levels of success from finding correct section values for small N*s, 
through finding how the section values increase going down in the table (i.e., the “y 
differences”), to stating a rule for computing the sections given the cuts (i.e., the func- 
tion). In our pilot classes, most 6^^ graders find the pattern of y differences for both 
minimum and maximum, and find and state the function for the minimum but not the 
maximum. When students do state a rule, it is usually in words (“just add 1 to the 
cuts”) and possibly including an instance (“say the cuts is 12, then one more is the 
sections”). In later lessons, students work through of a series of scaffolded situations 
(starting with single operator addition and going to combinations of two operators) to 
improve their ability to find patterns and to read and write verbal rules and algebraic 
expressions (see Figure 2). 

Rule writing is also practiced in the tutor. The Algebraic Expression Generation 
tutor unit (see Figure 3) is similar to early lessons in the Algebra Cognitive Tutor. 
Following the Action-Generalization principle and the results of the parametric study 
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Table 4. Comparisons of Situational and Abstract Problems in Five Content Areas 





Decimal place 
value 


Decimal arith 


Fraction addition 


Data interp- 
global 


Situation 


Show 5 
different ways 
that you can 
give Ben 
$4.07. [A place 
value table 
was provided 
to scaffold 
answers, but 
is not shown 
here] 


You had 
$8.72. Your 
grandmother 
gave you 
$25 for your 
birthday. How 
much money 
do you have 
now? 


Mrs. Jules 
bought each of 
her children a 
chocolate bar. 
Jarren ate 1/4 of 
a chocolate bar 
and Alicia ate 
1/5 of a chocolate 
bar. How much 
of a chocolate 
bar did they eat 
altogether? 


[2 scatterplots 
given] 

Do students sell 
more boxes of 
Candy Bars or 
Cookies as the 
months pass? 


% Correct 


61% 


65% 


32% 


62% 


Abstract 


List 5 different 
ways to show 
the amount 
4.07. [Place 
value table 
given.] 


Add: 8.72 + 
25 


Add: 1/4 +1/5 


[Scatterplots 
given] Are there 
more Moops 
per Zog in the 
Left graph 
or the Right 
graph? 


% Correct 


20% 


35% 


22% 


48% 





Data interpretation-local 


Factors & Multiples 


Situation 


[Scatterplot given] When 
9 players from Anchoville 
were present, how many 
pizzas got eaten? 


You work at a candy store. Your boss 
has asked you to figure out the different 
ways she could package the jelly beans 
and chocolate eggs, and she wants to 
know all the possible ways. If there are 
64 jelly beans and 40 chocolate eggs 
and she wants each package to be the 
same, what are the different numbers of 
packages vou could make? 


% Correct 


64% 


20% 


Abstract 


[Scatterplot given] In the 
Left graph, when there 
were 9 Pooks, how many 
Feeps were there? 


The common factors of 64 & 40 are: 


% Correct 


65% 


37% 
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To celabrsto the st^i ci sixth gracfe. your class is going to have a party. They are goir’ig u 
buy a rectangular sheet cake. The cake needs to be ^iced into pieces. 



Situation 1 

Your job Is to get the minimum niumber of pieces from each cut All ihe cuts must be a 
straight line and must start and erxi on a 



Complete the dh^ as you fir^sh each cake. Don't be timid; draw extra rectangles if you 
need them. 



Nimiber 


Mii^murrt nunib&f 


ctcuis 


ut sections 


1 


2 


2 




3 









Below are two examples of sin^e cut 
cakes that are iced. Notice that the 
sectkjns do not need to be even, just be 
certain that the lines are straight. 






Figure 1. Text excerpt from lesson 1.1 in the 2001-02 version of Cog- 
nitive Tutor Math 6. 



described above, students are first asked to solve for instances (rows 1 and 2 in Figure 
3) before writing a general expression (final row). A major difference in the 6^^ grade 
tutor is the addition of the “Show Your Work” column. Having students move from 
instances to the algebraic expression is consistent with the notion of algebra as a gen- 
eralization of arithmetic (e.g., Bednarz, Kieran, & Lee, 1996). Much to our surprise, 
however, we discovered that this generalization step is not the difficult one for stu- 
dents. In two studies reported on and referenced in Heffernan & Koedinger (1998), 
we found that students were not much worse at writing a general expression for story 
problems (49% correct) like “X * 2.0 - 1.25” for the problem in Figure 3 than writing 
an instance-based expression (53%) like “5 * 2.0 - 1.25”. However, writing either 
kind of expression was significantly more difficult than finding a numeric solution 
like 8.75 for the same story problems. In other words, it is the explanation step, going 
from “doing with an instance” to “explaining with an instance”, that is particularly 
difficult for students. Speaking in terms of students’ learning difficulties, one might 
say that algebra is the “explanation of arithmetic” rather than the “generalization of 
arithmetic”. 

As indicated above, the Situation-Abstraction principle is at work in multiple 
places in the curriculum. Some problem situations are more authentic to real world 
concerns, while others, like the Cake and Marble Machine problems, are more play- 
ful. How well individual situations work either in terms of motivating students or 
making ideas more accessible are open questions. The process of generating problem 
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Situation 3 Carol, feeling confident that she could break any code, went to the 
Marvelous; Marble Machjne. Unlike the coniputer code maker, the Marvelous Marine 
Machine uses tvyo operators to corripute Its secret riumber. 

The Marble Machine will take ordinary marbles and replace them with even.mpre beautifu 
colored marbles. Ofcoiirse there Is a catch! Yqumust break the code.iri order to keep ar 
marbles. If you cannot break the code, you will lose your marbles. 



The following table shows how many colorful marbles you will get when you insert ordinar 
marbles. Help Carol break the code. ' 



Ordihaiy Marbles 


Colorful Marbles 












26 







' ; 05 ■ ' ^ 503 - ;;;■ 

How do you get from 2 ordinary to 10 colorful marbles? 



33f H ^ 
A* ^ ^ 



b) Odes this sarhe procedure work from 3 ordinary mart)^ to 14 colorful maples? 
(If riot.try aripther possibility for the previous questic^^ 



Figure 2. Text excerpt showing an example pattern finding activity 
with student work. 






situations is primarily an art at this point, but the process of evaluating problem situa- 
tions (what situations are more accessible or motivating and why) could be more of a 
science than it is now. 

A key premise of the algebra strand is that it will be pedagogically more effec- 
tive if students first learn the syntax and semantics of the algebra language by writing 
and reading it before learning to use this language in problem solving. While writing 
and reading algebraic expressions is a regular activity throughout the grade cur- 
riculum, little emphasis is placed on using algebraic expressions in problem solving. 
Students frequently engage in algebraic problem solving, but it is supported using 
strategies other than equation solving. One such strategy involves the use of pictures 
as I describe in the next section. 

Picture Algebra lessons illustrating Visual-Verbal and Conceptual- 

Procedural. 

Figure 4 shows our initial instruction of Picture Algebra from Lesson 1.9 of the 
2(X)l-02 version of the text and Figure 5 shows an image from the associated tutor 
unit. This strategy is a variation of strategies used in Asian curricula (e.g., Singapore 
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Problem 



You went to the grocery store to buy bottles of the new green 
colored ketchup. Each bottle costs $2.00 and you had a coupon 
of $1.25 off your grocery bill. 

(1) If you bought 2 bottles , how much money would you owe? 

(2) If you bought S bottles , how much money would you owe? 

In the row labeled "Formula”, define a variable for the number 
of bottles and use that variable to write an expression that will 
allow you to calculate the money you owe. 



Problem FUNCTION-GEN- KETCH- UP 




Figure 3. Image of Algebraic Expression Generation tutor unit with 
“Show your work” column. 



Ministry of Education, 1999). It also bears similarity with picture strategies used in 
Math in Context. As one point of evidence for the power of this strategy, we have 
observed 6'^ graders using it to successfully solve problems that many older students 
(who usually attempt an equation solving strategy) fail to solve (Koedinger & Terao, 
2002). Whereas 7^^ graders in a sample from Bednarz & Janvier (1996, p. 120) were 
only 5% correct on the problem below, 6^ graders in a Cognitive Tutor class were 26% 
correct on this problem after Picture Algebra instruction. 

The use of Picture Algebra is consistent with the Visual- Verbal principle. This 
strategy integrates students* visual sense of quantity size with their prior verbal and 
symbolic skills for performing arithmetic operations. By drawing boxes of appropriate 
sizes, labeling, and performing associating arithmetic operations, students construct 
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Situation 1 

Suppose you want to buy a CD and magazine. The two cost $17 together If the CD costs 
$9 more than the magazine, how much does the magazine cost? 



One way to view this problem is to draw a picture to represent the objects. Be sure to label 
the drawings 



Item 

Cost of Magazine 
Cost of CD 




^ Total Cost 



If you remove the extra $9 from the total cost of $17, what will be left? 

Figure 4. Text excerpt showing initial instruction of Picture Algebra from 
Lesson 1.9 of the 2001-02 version. 



an integrated representation that may assist them in developing a “central conceptual 
structure” for algebraic reasoning. 

Picture Algebra is also consistent with the Conceptual-Procedural principle. It 
is meant to help students acquire a conceptual understanding of key algebraic ideas. 
One key idea regards transformations that maintain equality, for instance, subtracting 
9 from the bars on the left and the total (cf., Rittle-Johnson & Alibali, 1999). A second 
regards methods for recombining constraints on unknowns, for instance, seeing that 
the equal parts of the unknowns can be combined (analogous to transforming “.x + (.x + 
9)” into “2x + 9”). Currently, instruction that emphasizes integration of this conceptual 
approach with equation solving procedures is being left for and 8^^ grade in our cur- 
ricular program. It is a good question whether or not saving the integration for later is a 
violation of the Conceptual-Procedural principle and, separately, whether or not earlier 
integration would help learning. 

Examples from the Rational Number Strand of Cognitive I\itor Math 6 

Rational numbers are a cornerstone of advanced mathematics and a major stum- 
bling block for many children. Multiple units in the curriculum, especially Units 2-7, 
address number knowledge and rational numbers in particular. 

Decimal place value and arithmetic lessons illustrating Situation-Abstrac- 
tion and Conceptual-Procedural. 

In text unit 2 and associated Tbtor units, we targeted decimal number understand- 
ing with a particular emphasis on combining the Situation-Abstraction and the Con- 
ceptual-Procedural principles. The Conceptual-Procedural principle was employed 
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Figure 5. Image of problem from Picture Algebra tutor unit. In the 
Diagram window, students draw boxes of appropriate size and label 
them (the text is given but the boxes are not). In the “Show your work” 
window, students must justify their answers. The tutor monitors each step 
and provides context-sensitive hints only as needed (by clicking on the 
? icon). 



through the integration of place value concepts and decimal addition and subtraction 
procedures. The curriculum emphasizes the alignment and adding of equivalent place 
values (add ones to ones, tenths to tenths, etc.) as a rationale for the procedure of 
aligning the decimal point. The Situation-Abstraction principle was employed through 
the initial use of situations involving money quantities before other non-money and 
abstract situations. Money contexts appear to dramatically reduce decimal alignment 
errors in arithmetic, as we saw in Koedinger & Nathan (in press) and in the difficulty 
factors comparison illustrated above. The two principles combine well here as the key 
abstract conceptual-procedural link, do not add different place values, has an easier to 
understand analog in the concrete money situation, do not add dollars and cents. In 
other words, add ones to ones and tenths to tenths is analogous to add dollars to dol- 
lars and dimes to dimes. Kittle- Johnson & Koedinger (2002) report on an experiment 
demonstrating the value of iterating between conceptual and procedural lessons, first 
within the familiar money situations and then more abstractly. 
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Penny War Results for Tuesday 



Fraction concept and addition lessons illustrating Visual-Verbal and Action- 

Generalization. 

Learning to add and subtract fractions is particularly difficult for students as 
indicated by the fact that incorrect procedures persist into high school and reflect 
basic misunderstandings of fractions (e.g., Kouba, Carpenter, & Swafford, 1989). The 
most common error children make is to add the numerators and denominators (e.g., 
1/4 + 1/5 = 2/9), which violates basic part-whole concepts and can lead an answer 
that is smaller than either addend. In Rittle- Johnson & Koedinger (2001) we explored 
whether presenting problems with pictures or within a situational context would 
enhance performance and, in particular, provide a source for sense-making that would 
reduce this common error. Indeed, on a pre-test given to 6^^ graders, students per- 
formed better when given a problems with pictures (37%) or within a situation (32%; 
see Table 4) than when given an abstract number sentence (22%). In particular, the 
combine numerator and denominator 
error, while still common prior to 
instruction, was significantly lower 
for both pictures (38%) and situations 
(39%) than for the number sentence 
(48%). Relevant prior knowledge 
appears to be elicited by pictures and 
situations and it appears that, even 
without instruction, some students use 
such knowledge for sense making. 

In addition to indicating room 
for improvement among 6^^ graders, 
these results suggest promise for 
applying the Situation-Abstraction 
and Visual- Verbal principles to frac- 
tion concepts and fraction arithmetic. 

Lesson 5.10 addressed fraction addi- 
tion, but it is important to first review 
what ideas were addressed earlier 
in the unit along the developmental 
corridor leading to fraction addition. 

Students are encouraged to discover 
multiple ways of expressing frac- 
tions of equivalent value. Follow- 
ing the Visual-Verbal principle, the 
concept of fraction equivalence (and 
comparison) is supported through 




Figure 6. Student using her fraction strip to 
measure the result of combining the 1/3 of 
the weekly goal collected on Monday and 
the 1/6 of the weekly goal on Tuesday. She 
measures the result to be 1/2. 
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measurement activities and pictorial representations — fraction bars, number lines, and 
analogous representations in the Cognitive Tutor software. As part of the measurement 
activities, students create a fraction ruler that is a whole unit in length and labeled with 
multiple fractions. This ruler is both a frequently used tool and a concrete reminder of 
fraction equivalence as many positions on the ruler are labeled with multiple fractions 
(e.g., 1/4 and 2/8). 

In lesson 5.1, students begin an activity involving a fund raising effort at the 
middle school (see Figure 6). (Note: “Penny Wars” is the name of an annual fund- 
raiser at a local school, but we are changing the name in the text to “Penny Race”.) 
The fund-raiser has a goal for the week and students record in a bar chart the fractional 
progress toward that goal for each grade on each day. As illustrated in Figure 6, stu- 
dents use their fraction ruler to measure the total of the 7^^ graders’ penny collection 
on Monday, 1/3 of the goal, and on Tuesday, 1/6 of the goal. The measurement shows 
that the graders are now 1/2 of the way to their goal. 

At this point, the Action-Generalization principle is employed. Students record 
such measurements (actions) and instances of associated arithmetic facts (e.g., 1/3 + 
1/6 = 1/2). As shown in Figure 7, students revisit these facts in lesson 5.10 and have 
a discussion about how to add fractions. Why is it that 1/3 + 1/6 is 1/2? The prior 
emphasis on fraction equivalence and its reification on the fraction ruler has the con- 
sequence that it is not a difficult leap for students to think of 1/3 as 2/6. Thus, they 
are in position to recognize that they already know how to add 2/6 + 1/6 (like fraction 
addition is a topic prior to 6^*^ grade). Having begun to recognize a pattern, students are 
asked to experiment with their procedure idea on other arithmetic facts. Once their 



Situation 2 - Adding Fractions with Unlike Denominators 

Look back at the penny war results for Tuesday (page 7^. The sixth grade collected 1M of its 
weekly goal on Monday, and 1/8 of its goal on Tuesday. The total for the week was 
measured at 3/8 of the total goal. The diagram and number line display the problem and the 



answer. 

1 ^ 

4 8 




H — I — ^ ^ ^ ^ — \ — ^ 

1 3 

° 4 8 

Figure 7. Revisiting the fraction addition facts determined by physical 
arithmetic to use as a basis for finding patterns and writing a fraction 
addition procedure. 
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Figure 8. A student’s procedure for addition of unlike fractions. 

procedure seems to be working they write it down. Figure 8 shows one student’s writ- 
ten procedure. 

Note the progression of activities in this the application of the Action-General- 
ization principle. First students are “doing with instances” by using measurement to 
physically find fraction sums. Next, students are “doing with a generalization” by 
attempting to test an idea for a fraction addition procedure (which is not yet articu- 
lated) with new instances. Finally students “explain with a generalization” by writing 
a procedure for fraction addition in their own words. This approach is applied for other 
concepts in the curriculum. 

An Initial Full Course Evaluation of Cognitive I\itor Math 6 

We began research and development of Cognitive Tutor Math 6 in 1999 and 
immediately began testing in classrooms. In the 2000-2001 school year, a complete 
alpha version of Cognitive Tutor Math 6 was used at two Pittsburgh-area schools. To 
evaluate the effect of the course on raising student achievement, we contrasted end- 
of-course performance of students in Cognitive Tutor classes with students in schools 
with comparable demographics. Two types of assessments were used, a “Standard- 
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ized Skills” test made up of selected test items from State and National standardized 
tests and a “Problem Solving and Concepts” test that we developed to assess students' 
problem solving abilities and conceptual understanding. The test items were selected 
or created to be challenging and with the goal that average performance would not be 
far from 50%. 

Table 5 summarizes students’ average percent correct on the two assessments in 
Cognitive Tutor and comparison classes in two school districts. All four differences 
between Cognitive Tutor and comparison student performance are statistically reli- 
able. 

It was not feasible to randomly assign students or teachers to tutor versus com- 
parison classes given the logistics of agreements with the districts. Thus, there is some 
chance that the differences are due to other factors. However, it is encouraging that 
students in Cognitive Tutor classes consistently outperformed students in comparison 
classes on both assessments and at both schools. 

Conclusions and Future Work 

Cognitive Tutor courses have achieved considerable success both in terms of 
raising student achievement and in reaching large numbers of students. So have other 
research-based mathematics curricula. Nevertheless, it is critical we remain humble 
about our theories and beliefs about appropriate and effective mathematics instruction. 
There is still plenty of room for novel ideas in mathematics instruction. Moreover, 
there is even more room for firm evidence that existing ideas are effective in increas- 
ing student learning and do so in ways that can scale up to widespread cost-effective 
use. A particular gap exists between theories of general cognitive function and theories 
of mathematics content. We need theories and principles in this intermediate terrain 
that provide pedagogical leverage. Ma’s (1999) work and the more general effort to 
characterize pedagogical content knowledge (e.g., Shulman, 1986) are extremely 
valuable toward filling the gap, but better still if such work makes contact with theo- 
ries of general cognitive function. From the psychological side, efforts to characterize 
general learning and instructional principles (Bransford, et al., 1999) are also useful. 
However, better still if we had more specific principles and associated methods that 
make contact with specific pedagogical content issues. 



Table 5. Results of an Initial Full Course Evaluation of Cognitive Tutor Math 6 



School 

District 


Test 


Cognitive 

Tutor 


Comparison 

Classes 


T-test 
p- value 


District A 


Standardized 


63% (N=47) 


54% (N=50) 


<0.001 




Solving & Concepts 


50% (N=48) 


41% (N=48) 


0.007 


District B 


Standardized 


71% (N=37) 


64% (N=132) 


0.046 




Solving & Concepts 


69% (N=38) 


62%(N=116) 


0.045 
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Toward evidence for instructional design principles 



I have made some, I hope humble, attempts to articulate some principles within 
this intermediate terrain: Situation-Abstraction, Action-Generalization, Visual-Verbal, 
and Conceptual-Procedural. They bear considerable similarity to a number of other 
approaches that I am aware of and probably many more that I am not. My point is not 
to argue that these principles are particularly new. Nor is it to argue that I have pro- 
vided firm proof for any them. Indeed, I think attempts to “prove” such principles are 
premature and perhaps even wrong-headed. I do suggest that we need to accumulate 
evidence that indicates guidelines for when and how to apply principles like these. I 
also suggest that such principles cannot be applied purely analytically - they require 
associated empirical facts, for instance, about whether particular situations or visual- 
izations ease understanding. 

This paper presented an overview of our new Cognitive Tutor Math 6 course, 
some samples of text and tutor activities within it, and some early evaluation data that 
seems to indicate that students may be learning more from this course than alterna- 
tives. I actually have not said much about the unique features of Cognitive Tutors, but 
have written about them elsewhere. Perhaps the most important feature is the capabil- 
ity to approximate the kind of moment-by-moment individualized assistance that a 
good human tutor can provide. The classroom teacher is a critical component in Cog- 
nitive Tutor classes, but when teachers are in the computer lab they have the support 
of what is, in effect, an automated teacher’s aid for every student. This extra support 
allows them to give more individualized attention to the students who most need it 
(Schofield, Evans-Rhodes, & Huber, 1990; Wertheimer, 1990). Such automated sup- 
port is not possible without theory-based analyses of learners and learning. 

Looking forward, I would like to see mathematics educators and cognitive scien- 
tists explore the feasibility and advisability of a large program of research, on the order 
of the human genome project, to identify and catalog the knowledge structures or cog- 
nitive objects that underlie mathematical thinking and learning. This knowledge base 
would help drive new instructional innovations and provide for a rich cause theory of 
how, when, & why they may work. Like the human genome project, such an ambitious 
program would yield major applied and basic research outcomes. 
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